{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# Python爬虫常用库详细教程\n",
    "\n",
    "## 1. Requests：HTTP for Humans\n",
    "\n",
    "### 1.1 简介\n",
    "\n",
    "Requests是一个优雅而简单的Python HTTP库，被称为\"HTTP for Humans\"。它抽象了复杂的底层网络细节，使得发送HTTP/1.1请求变得异常简单。\n",
    "\n",
    "### 1.2 安装\n",
    "\n",
    "使用pip安装Requests：\n",
    "\n",
    "```bash\n",
    "pip install requests\n",
    "```\n",
    "\n",
    "### 1.3 基本用法\n",
    "\n",
    "#### 1.3.1 发送GET请求\n",
    "\n",
    "```python\n",
    "import requests\n",
    "\n",
    "# 简单的GET请求\n",
    "response = requests.get('https://api.github.com/events')\n",
    "print(response.status_code)  # 200\n",
    "print(response.text)  # 打印响应内容\n",
    "\n",
    "# 带参数的GET请求\n",
    "payload = {'key1': 'value1', 'key2': 'value2'}\n",
    "response = requests.get('https://httpbin.org/get', params=payload)\n",
    "print(response.url)  # https://httpbin.org/get?key1=value1&key2=value2\n",
    "```\n",
    "\n",
    "#### 1.3.2 发送POST请求\n",
    "\n",
    "```python\n",
    "# 发送表单数据\n",
    "payload = {'key1': 'value1', 'key2': 'value2'}\n",
    "response = requests.post('https://httpbin.org/post', data=payload)\n",
    "print(response.text)\n",
    "\n",
    "# 发送JSON数据\n",
    "import json\n",
    "response = requests.post('https://api.github.com/some/endpoint', data=json.dumps(payload))\n",
    "```\n",
    "\n",
    "### 1.4 处理响应\n",
    "\n",
    "```python\n",
    "response = requests.get('https://api.github.com/events')\n",
    "\n",
    "# 检查响应状态码\n",
    "if response.status_code == 200:\n",
    "    print('请求成功!')\n",
    "\n",
    "# 获取响应内容\n",
    "print(response.text)  # 文本形式\n",
    "print(response.content)  # 字节形式\n",
    "print(response.json())  # JSON形式（如果响应是JSON格式）\n",
    "\n",
    "# 获取响应头\n",
    "print(response.headers)\n",
    "\n",
    "# 获取编码\n",
    "print(response.encoding)\n",
    "```\n",
    "\n",
    "### 1.5 处理Cookies\n",
    "\n",
    "```python\n",
    "# 发送cookies\n",
    "cookies = dict(cookies_are='working')\n",
    "response = requests.get('https://httpbin.org/cookies', cookies=cookies)\n",
    "\n",
    "# 获取cookies\n",
    "print(response.cookies['example_cookie_name'])\n",
    "\n",
    "# 使用Session保持cookies\n",
    "session = requests.Session()\n",
    "session.get('https://httpbin.org/cookies/set/sessioncookie/123456789')\n",
    "response = session.get('https://httpbin.org/cookies')\n",
    "print(response.text)  # 将显示服务器端的cookie\n",
    "```\n",
    "\n",
    "### 1.6 处理超时和异常\n",
    "\n",
    "```python\n",
    "try:\n",
    "    response = requests.get('https://api.github.com', timeout=1)\n",
    "    response.raise_for_status()  # 如果状态不是200，将抛出HTTPError异常\n",
    "except requests.exceptions.Timeout:\n",
    "    print(\"请求超时\")\n",
    "except requests.exceptions.HTTPError as err:\n",
    "    print(f\"HTTP错误occurred: {err}\")\n",
    "except requests.exceptions.RequestException as err:\n",
    "    print(f\"发生错误: {err}\")\n",
    "```\n",
    "\n",
    "### 1.7 自定义请求头\n",
    "\n",
    "```python\n",
    "headers = {\n",
    "    'User-Agent': 'MyAwesomeBot/1.0',\n",
    "    'Accept': 'application/json'\n",
    "}\n",
    "response = requests.get('https://api.github.com', headers=headers)\n",
    "```\n",
    "\n",
    "### 1.8 处理身份验证\n",
    "\n",
    "```python\n",
    "# 基本身份验证\n",
    "response = requests.get('https://api.github.com/user', auth=('username', 'password'))\n",
    "\n",
    "# OAuth认证\n",
    "headers = {'Authorization': 'token YOUR_OAUTH_TOKEN'}\n",
    "response = requests.get('https://api.github.com/user', headers=headers)\n",
    "```\n",
    "\n",
    "### 1.9 高级功能：代理和SSL验证\n",
    "\n",
    "```python\n",
    "# 使用代理\n",
    "proxies = {\n",
    "  'http': 'http://10.10.1.10:3128',\n",
    "  'https': 'http://10.10.1.10:1080',\n",
    "}\n",
    "response = requests.get('http://example.org', proxies=proxies)\n",
    "\n",
    "# 禁用SSL证书验证（不推荐在生产环境中使用）\n",
    "response = requests.get('https://api.github.com', verify=False)\n",
    "```\n",
    "\n",
    "### 1.10 最佳实践和注意事项\n",
    "\n",
    "1. 始终检查响应状态码。\n",
    "\n",
    "2. 使用 `with` 语句来自动关闭响应：\n",
    "\n",
    "   ```python\n",
    "   with requests.get('https://api.github.com/events') as response:\n",
    "       print(response.text)\n",
    "   ```\n",
    "\n",
    "3. 对于需要多次请求的情况，使用 `Session` 对象可以提高性能。\n",
    "\n",
    "4. 注意处理大文件时的内存使用：\n",
    "\n",
    "   ```python\n",
    "   with requests.get('https://example.com/large-file', stream=True) as response:\n",
    "       with open('large-file.zip', 'wb') as f:\n",
    "           for chunk in response.iter_content(chunk_size=8192): \n",
    "               f.write(chunk)\n",
    "   ```\n",
    "\n",
    "5. 遵守网站的robots.txt规则和使用条款。\n",
    "\n",
    "Requests库是Python爬虫开发中不可或缺的工具。它简化了HTTP请求的发送和处理过程，使得开发者可以专注于数据的获取和处理，而不必过多关注底层的网络细节。在实际的爬虫项目中，Requests通常是与其他解析库（如Beautiful Soup或lxml）配合使用的。\n",
    "\n",
    "## 2. Beautiful Soup：优雅地解析HTML和XML\n",
    "\n",
    "### 2.1 简介\n",
    "\n",
    "Beautiful Soup是一个用于从HTML和XML文件中提取数据的Python库。它能够通过定位HTML标签来组织复杂的网络信息，为网络爬虫提供了便利的数据提取方式。\n",
    "\n",
    "### 2.2 安装\n",
    "\n",
    "使用pip安装Beautiful Soup：\n",
    "\n",
    "```bash\n",
    "pip install beautifulsoup4\n",
    "```\n",
    "\n",
    "注意：Beautiful Soup支持多种解析器，推荐同时安装lxml解析器：\n",
    "\n",
    "```bash\n",
    "pip install lxml\n",
    "```\n",
    "\n",
    "### 2.3 基本用法\n",
    "\n",
    "#### 2.3.1 创建Beautiful Soup对象\n",
    "\n",
    "```python\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# 从HTML字符串创建\n",
    "html_doc = \"\"\"\n",
    "<html><head><title>The Dormouse's story</title></head>\n",
    "<body>\n",
    "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
    "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
    "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
    "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
    "and they lived at the bottom of a well.</p>\n",
    "<p class=\"story\">...</p>\n",
    "\"\"\"\n",
    "\n",
    "soup = BeautifulSoup(html_doc, 'lxml')\n",
    "\n",
    "# 从文件创建\n",
    "with open(\"index.html\") as f:\n",
    "    soup = BeautifulSoup(f, 'lxml')\n",
    "```\n",
    "\n",
    "#### 2.3.2 遍历文档树\n",
    "\n",
    "```python\n",
    "# 获取标题\n",
    "print(soup.title)  # <title>The Dormouse's story</title>\n",
    "print(soup.title.name)  # title\n",
    "print(soup.title.string)  # The Dormouse's story\n",
    "\n",
    "# 获取段落\n",
    "print(soup.p)  # <p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "\n",
    "# 获取链接\n",
    "print(soup.a)  # <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>\n",
    "\n",
    "# 获取所有链接\n",
    "print(soup.find_all('a'))\n",
    "\n",
    "# 获取文本内容\n",
    "print(soup.get_text())\n",
    "```\n",
    "\n",
    "#### 2.3.3 搜索树\n",
    "\n",
    "##### 1. 按标签名搜索\n",
    "\n",
    "```python\n",
    "soup.find_all('a')\n",
    "```\n",
    "\n",
    "##### 2. 按CSS类搜索\n",
    "\n",
    "```python\n",
    "soup.find_all(\"p\", class_=\"story\")\n",
    "```\n",
    "\n",
    "##### 3. 按ID搜索\n",
    "\n",
    "```python\n",
    "soup.find(id=\"link1\")\n",
    "```\n",
    "\n",
    "##### 4. 按属性搜索\n",
    "\n",
    "```python\n",
    "soup.find_all(href=re.compile(\"elsie\"))\n",
    "```\n",
    "\n",
    "##### 5. 使用CSS选择器\n",
    "\n",
    "```python\n",
    "soup.select(\"p.story > a\")\n",
    "```\n",
    "\n",
    "### 2.4 修改文档树\n",
    "\n",
    "#### 2.4.1 修改标签名\n",
    "\n",
    "```python\n",
    "tag = soup.b\n",
    "tag.name = \"blockquote\"\n",
    "```\n",
    "\n",
    "#### 2.4.2 修改属性\n",
    "\n",
    "```python\n",
    "tag['class'] = 'verybold'\n",
    "tag['id'] = 1\n",
    "```\n",
    "\n",
    "#### 2.4.3 修改文本\n",
    "\n",
    "```python\n",
    "tag.string = \"New text content\"\n",
    "```\n",
    "\n",
    "#### 2.4.4 添加内容\n",
    "\n",
    "```python\n",
    "soup.body.append(BeautifulSoup(\"<p>New paragraph</p>\", 'lxml'))\n",
    "```\n",
    "\n",
    "### 2.5 输出\n",
    "\n",
    "#### 2.5.1 格式化输出\n",
    "\n",
    "```python\n",
    "print(soup.prettify())\n",
    "```\n",
    "\n",
    "#### 2.5.2 指定编码\n",
    "\n",
    "```python\n",
    "print(soup.prettify(encoding='latin-1'))\n",
    "```\n",
    "\n",
    "### 2.6 高级用法\n",
    "\n",
    "#### 2.6.1 使用CSS选择器\n",
    "\n",
    "Beautiful Soup支持大部分CSS选择器：\n",
    "\n",
    "```python\n",
    "soup.select(\"div.class_name\")  # 选择所有class为class_name的div元素\n",
    "soup.select(\"#id_name\")  # 选择id为id_name的元素\n",
    "soup.select(\"div > p\")  # 选择div直接子元素中的p元素\n",
    "```\n",
    "\n",
    "#### 2.6.2 处理文档编码\n",
    "\n",
    "Beautiful Soup会自动检测文档的编码，但你也可以指定编码：\n",
    "\n",
    "```python\n",
    "soup = BeautifulSoup(html_doc, 'lxml', from_encoding=\"iso-8859-8\")\n",
    "```\n",
    "\n",
    "#### 2.6.3 解析部分文档\n",
    "\n",
    "有时你可能只想解析文档的一部分：\n",
    "\n",
    "```python\n",
    "soup = BeautifulSoup(\"<a><b>text1</b><c>text2</c></b></a>\", 'lxml')\n",
    "soup.b.find_all('c')  # 只在<b>标签内搜索\n",
    "```\n",
    "\n",
    "### 2.7 性能考虑\n",
    "\n",
    "1. 选择合适的解析器：lxml解析器通常比内置解析器更快。\n",
    "\n",
    "2. 使用 `SoupStrainer` 只解析你需要的文档部分：\n",
    "\n",
    "   ```python\n",
    "   from bs4 import SoupStrainer\n",
    "   \n",
    "   only_a_tags = SoupStrainer(\"a\")\n",
    "   soup = BeautifulSoup(html_doc, 'lxml', parse_only=only_a_tags)\n",
    "   ```\n",
    "\n",
    "3. 对于大文件，考虑使用 `yield_per()` 方法逐步处理。\n",
    "\n",
    "### 2.8 常见问题和解决方案\n",
    "\n",
    "1. 编码问题：使用 `from_encoding` 参数指定正确的编码。\n",
    "2. 解析错误：尝试使用不同的解析器（如html.parser, lxml, html5lib）。\n",
    "3. 性能问题：对于大文件，考虑使用流式解析或其他更高效的解析方法。\n",
    "\n",
    "Beautiful Soup是一个强大而灵活的库，特别适合处理结构不太规范的HTML。它与Requests库配合使用，可以轻松地从网页中提取所需的信息。然而，对于大规模爬虫项目或需要高性能的场景，可能需要考虑使用lxml等更快的解析器。\n",
    "\n",
    "## 3. lxml：高效的XML和HTML解析器\n",
    "\n",
    "### 3.1 简介\n",
    "\n",
    "lxml是一个高性能的XML和HTML解析库，它结合了libxml2和libxslt的优点。lxml不仅速度快，而且具有友好的API，支持XPath、XSLT等功能。\n",
    "\n",
    "### 3.2 安装\n",
    "\n",
    "使用pip安装lxml：\n",
    "\n",
    "```bash\n",
    "pip install lxml\n",
    "```\n",
    "\n",
    "### 3.3 基本用法\n",
    "\n",
    "#### 3.3.1 解析HTML\n",
    "\n",
    "```python\n",
    "from lxml import etree\n",
    "\n",
    "# 从字符串解析HTML\n",
    "html = '''\n",
    "<html>\n",
    "    <body>\n",
    "        <h1>My First Heading</h1>\n",
    "        <p>My first paragraph.</p>\n",
    "    </body>\n",
    "</html>\n",
    "'''\n",
    "tree = etree.HTML(html)\n",
    "\n",
    "# 从文件解析HTML\n",
    "tree = etree.parse('example.html', etree.HTMLParser())\n",
    "```\n",
    "\n",
    "#### 3.3.2 解析XML\n",
    "\n",
    "```python\n",
    "from lxml import etree\n",
    "\n",
    "# 从字符串解析XML\n",
    "xml = '''\n",
    "<bookstore>\n",
    "    <book category=\"cooking\">\n",
    "        <title lang=\"en\">Everyday Italian</title>\n",
    "        <author>Giada De Laurentiis</author>\n",
    "        <year>2005</year>\n",
    "        <price>30.00</price>\n",
    "    </book>\n",
    "</bookstore>\n",
    "'''\n",
    "tree = etree.XML(xml)\n",
    "\n",
    "# 从文件解析XML\n",
    "tree = etree.parse('example.xml')\n",
    "```\n",
    "\n",
    "### 3.4 使用XPath\n",
    "\n",
    "XPath是XML路径语言，用于在XML文档中查找信息。lxml对XPath有很好的支持。\n",
    "\n",
    "```python\n",
    "# 选择所有book元素\n",
    "books = tree.xpath('//book')\n",
    "\n",
    "# 选择所有title元素\n",
    "titles = tree.xpath('//title/text()')\n",
    "\n",
    "# 选择category属性为'cooking'的book元素\n",
    "cooking_books = tree.xpath(\"//book[@category='cooking']\")\n",
    "\n",
    "# 选择lang属性为'en'的title元素的文本\n",
    "english_titles = tree.xpath(\"//title[@lang='en']/text()\")\n",
    "```\n",
    "\n",
    "### 3.5 使用CSS选择器\n",
    "\n",
    "lxml也支持CSS选择器，这对于习惯于使用CSS的开发者来说非常方便。\n",
    "\n",
    "```python\n",
    "from lxml.cssselect import CSSSelector\n",
    "\n",
    "# 选择所有p元素\n",
    "selector = CSSSelector('p')\n",
    "paragraphs = selector(tree)\n",
    "\n",
    "# 选择class为'highlight'的元素\n",
    "highlights = tree.cssselect('.highlight')\n",
    "```\n",
    "\n",
    "### 3.6 创建和修改文档\n",
    "\n",
    "#### 3.6.1 创建新元素\n",
    "\n",
    "```python\n",
    "root = etree.Element(\"root\")\n",
    "child = etree.SubElement(root, \"child\")\n",
    "child.text = \"Some text\"\n",
    "child.set(\"attribute\", \"value\")\n",
    "\n",
    "# 将树转换为字符串\n",
    "xml_string = etree.tostring(root, pretty_print=True).decode()\n",
    "print(xml_string)\n",
    "```\n",
    "\n",
    "#### 3.6.2 修改元素\n",
    "\n",
    "```python\n",
    "# 修改文本\n",
    "child.text = \"New text\"\n",
    "\n",
    "# 添加/修改属性\n",
    "child.set(\"new_attribute\", \"new_value\")\n",
    "\n",
    "# 删除属性\n",
    "child.attrib.pop(\"attribute\", None)\n",
    "\n",
    "# 删除元素\n",
    "root.remove(child)\n",
    "```\n",
    "\n",
    "### 3.7 序列化和反序列化\n",
    "\n",
    "#### 3.7.1 将树序列化为字符串\n",
    "\n",
    "```python\n",
    "xml_string = etree.tostring(tree, pretty_print=True, encoding='unicode')\n",
    "```\n",
    "\n",
    "#### 3.7.2 将字符串解析为树\n",
    "\n",
    "```python\n",
    "tree = etree.fromstring(xml_string)\n",
    "```\n",
    "\n",
    "### 3.8 使用XSLT转换XML\n",
    "\n",
    "XSLT（eXtensible Stylesheet Language Transformations）是一种用于将XML文档转换成其他XML文档的语言。lxml支持XSLT转换。\n",
    "\n",
    "```python\n",
    "from lxml import etree\n",
    "\n",
    "# XML数据\n",
    "xml = '''\n",
    "<employees>\n",
    "  <employee>\n",
    "    <name>John</name>\n",
    "    <age>30</age>\n",
    "  </employee>\n",
    "  <employee>\n",
    "    <name>Jane</name>\n",
    "    <age>25</age>\n",
    "  </employee>\n",
    "</employees>\n",
    "'''\n",
    "\n",
    "# XSLT样式表\n",
    "xslt = '''\n",
    "<xsl:stylesheet version=\"1.0\" xmlns:xsl=\"http://www.w3.org/1999/XSL/Transform\">\n",
    "  <xsl:template match=\"/\">\n",
    "    <html>\n",
    "      <body>\n",
    "        <h2>Employees</h2>\n",
    "        <table border=\"1\">\n",
    "          <tr>\n",
    "            <th>Name</th>\n",
    "            <th>Age</th>\n",
    "          </tr>\n",
    "          <xsl:for-each select=\"employees/employee\">\n",
    "          <tr>\n",
    "            <td><xsl:value-of select=\"name\"/></td>\n",
    "            <td><xsl:value-of select=\"age\"/></td>\n",
    "          </tr>\n",
    "          </xsl:for-each>\n",
    "        </table>\n",
    "      </body>\n",
    "    </html>\n",
    "  </xsl:template>\n",
    "</xsl:stylesheet>\n",
    "'''\n",
    "\n",
    "# 解析XML和XSLT\n",
    "xml_tree = etree.fromstring(xml)\n",
    "xslt_tree = etree.fromstring(xslt)\n",
    "\n",
    "# 创建XSLT转换器\n",
    "transform = etree.XSLT(xslt_tree)\n",
    "\n",
    "# 应用转换\n",
    "result_tree = transform(xml_tree)\n",
    "\n",
    "# 输出结果\n",
    "print(etree.tostring(result_tree, pretty_print=True).decode())\n",
    "```\n",
    "\n",
    "### 3.9 性能优化技巧\n",
    "\n",
    "1. 使用 `lxml.etree.iterparse()` 处理大文件：\n",
    "\n",
    "   ```python\n",
    "   for event, elem in etree.iterparse(\"large.xml\", tag=\"book\"):\n",
    "       print(elem.tag, elem.text)\n",
    "       elem.clear()\n",
    "   ```\n",
    "\n",
    "2. 使用XPath编译器提高重复查询的性能：\n",
    "\n",
    "   ```python\n",
    "   xpath_compiler = etree.XPath(\"//book[@category='fiction']\")\n",
    "   fiction_books = xpath_compiler(tree)\n",
    "   ```\n",
    "\n",
    "3. 使用 `lxml.etree.XMLParser(recover=True)` 处理错误的XML：\n",
    "\n",
    "   ```python\n",
    "   parser = etree.XMLParser(recover=True)\n",
    "   tree = etree.parse(\"faulty.xml\", parser)\n",
    "   ```\n",
    "\n",
    "### 3.10 lxml vs Beautiful Soup\n",
    "\n",
    "- 速度：lxml通常比Beautiful Soup快得多。\n",
    "- 内存使用：lxml的内存使用通常较低。\n",
    "- 功能：lxml提供了更多的XML特性支持。\n",
    "- 易用性：Beautiful Soup的API可能更直观，特别是对于初学者。\n",
    "- 解析能力：Beautiful Soup在处理非标准HTML时可能更宽容。\n",
    "\n",
    "选择使用lxml还是Beautiful Soup取决于你的具体需求。对于大规模爬虫项目或需要处理大量XML数据的情况，lxml是更好的选择。对于小型项目或处理非常混乱的HTML，Beautiful Soup可能更合适。\n",
    "\n",
    "## 4. Selenium：Web应用程序测试工具\n",
    "\n",
    "### 4.1 简介\n",
    "\n",
    "Selenium是一个用于Web应用程序测试的工具。它直接驱动浏览器，模拟用户操作，能够抓取JavaScript渲染的页面，因此在爬虫领域也被广泛使用。\n",
    "\n",
    "### 4.2 安装\n",
    "\n",
    "使用pip安装Selenium：\n",
    "\n",
    "```bash\n",
    "pip install selenium\n",
    "```\n",
    "\n",
    "你还需要安装对应浏览器的WebDriver。以Chrome为例，你需要下载ChromeDriver并将其添加到系统PATH中。\n",
    "\n",
    "### 4.3 基本用法\n",
    "\n",
    "#### 4.3.1 启动浏览器\n",
    "\n",
    "```python\n",
    "from selenium import webdriver\n",
    "\n",
    "# 启动Chrome浏览器\n",
    "driver = webdriver.Chrome()\n",
    "\n",
    "# 访问网页\n",
    "driver.get(\"https://www.example.com\")\n",
    "\n",
    "# 获取页面标题\n",
    "print(driver.title)\n",
    "\n",
    "# 关闭浏览器\n",
    "driver.quit()\n",
    "```\n",
    "\n",
    "#### 4.3.2 定位元素\n",
    "\n",
    "Selenium提供了多种方法来定位元素：\n",
    "\n",
    "```python\n",
    "# 通过ID定位\n",
    "element = driver.find_element_by_id(\"elementId\")\n",
    "\n",
    "# 通过名称定位\n",
    "element = driver.find_element_by_name(\"elementName\")\n",
    "\n",
    "# 通过XPath定位\n",
    "element = driver.find_element_by_xpath(\"//input[@name='password']\")\n",
    "\n",
    "# 通过CSS选择器定位\n",
    "element = driver.find_element_by_css_selector(\"#submit-button\")\n",
    "\n",
    "# 通过链接文本定位\n",
    "element = driver.find_element_by_link_text(\"Click here\")\n",
    "\n",
    "# 使用WebDriverWait等待元素出现\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "\n",
    "element = WebDriverWait(driver, 10).until(\n",
    "    EC.presence_of_element_located((By.ID, \"myDynamicElement\"))\n",
    ")\n",
    "```\n",
    "\n",
    "#### 4.3.3 操作元素\n",
    "\n",
    "```python\n",
    "# 点击元素\n",
    "element.click()\n",
    "\n",
    "# 输入文本\n",
    "element.send_keys(\"Hello, World!\")\n",
    "\n",
    "# 清除输入\n",
    "element.clear()\n",
    "\n",
    "# 提交表单\n",
    "element.submit()\n",
    "```\n",
    "\n",
    "#### 4.3.4 获取元素信息\n",
    "\n",
    "```python\n",
    "# 获取元素文本\n",
    "text = element.text\n",
    "\n",
    "# 获取属性值\n",
    "attribute = element.get_attribute(\"class\")\n",
    "\n",
    "# 检查元素是否可见\n",
    "is_visible = element.is_displayed()\n",
    "```\n",
    "\n",
    "### 4.4 处理动态内容\n",
    "\n",
    "#### 4.4.1 等待页面加载完成\n",
    "\n",
    "```python\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "from selenium.webdriver.common.by import By\n",
    "\n",
    "# 等待特定元素出现\n",
    "element = WebDriverWait(driver, 10).until(\n",
    "    EC.presence_of_element_located((By.ID, \"myDynamicElement\"))\n",
    ")\n",
    "\n",
    "# 等待页面标题包含特定文本\n",
    "WebDriverWait(driver, 10).until(EC.title_contains(\"Finished\"))\n",
    "```\n",
    "\n",
    "#### 4.4.2 执行JavaScript\n",
    "\n",
    "```python\n",
    "# 执行JavaScript脚本\n",
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "\n",
    "# 获取JavaScript返回值\n",
    "title = driver.execute_script(\"return document.title;\")\n",
    "```\n",
    "\n",
    "### 4.5 处理弹出窗口和框架\n",
    "\n",
    "#### 4.5.1 处理警告框\n",
    "\n",
    "```python\n",
    "from selenium.webdriver.common.alert import Alert\n",
    "\n",
    "# 切换到警告框\n",
    "alert = Alert(driver)\n",
    "\n",
    "# 获取警告框文本\n",
    "print(alert.text)\n",
    "\n",
    "# 接受警告框\n",
    "alert.accept()\n",
    "\n",
    "# 取消警告框\n",
    "alert.dismiss()\n",
    "```\n",
    "\n",
    "#### 4.5.2 处理框架\n",
    "\n",
    "```python\n",
    "# 切换到框架\n",
    "driver.switch_to.frame(\"frame_name\")\n",
    "\n",
    "# 切换回主文档\n",
    "driver.switch_to.default_content()\n",
    "```\n",
    "\n",
    "### 4.6 高级技巧\n",
    "\n",
    "#### 4.6.1 无头模式\n",
    "\n",
    "无头模式允许Selenium在后台运行浏览器，不显示GUI。\n",
    "\n",
    "```python\n",
    "from selenium.webdriver.chrome.options import Options\n",
    "\n",
    "chrome_options = Options()\n",
    "chrome_options.add_argument(\"--headless\")\n",
    "driver = webdriver.Chrome(options=chrome_options)\n",
    "```\n",
    "\n",
    "#### 4.6.2 管理Cookies\n",
    "\n",
    "```python\n",
    "# 添加Cookie\n",
    "driver.add_cookie({\"name\": \"key\", \"value\": \"value\"})\n",
    "\n",
    "# 获取所有Cookies\n",
    "print(driver.get_cookies())\n",
    "\n",
    "# 删除特定Cookie\n",
    "driver.delete_cookie(\"cookie_name\")\n",
    "\n",
    "# 删除所有Cookies\n",
    "driver.delete_all_cookies()\n",
    "```\n",
    "\n",
    "#### 4.6.3 截图\n",
    "\n",
    "```python\n",
    "# 截取整个页面\n",
    "driver.save_screenshot(\"screenshot.png\")\n",
    "\n",
    "# 截取特定元素\n",
    "element = driver.find_element_by_id(\"myElement\")\n",
    "element.screenshot(\"element_screenshot.png\")\n",
    "```\n",
    "\n",
    "### 4.7 最佳实践\n",
    "\n",
    "1. 使用显式等待而不是隐式等待。\n",
    "2. 在测试结束时始终调用 `driver.quit()` 以释放资源。\n",
    "3. 使用Page Object Model设计模式组织你的代码。\n",
    "4. 对于大规模爬虫，考虑使用Selenium Grid进行分布式测试。\n",
    "5. 定期更新WebDriver和Selenium库以获取最新的功能和bug修复。\n",
    "\n",
    "Selenium是一个强大的工具，特别适合需要与网页交互或处理动态内容的爬虫任务。然而，它比纯HTTP请求的方法要慢，所以在设计爬虫时要权衡性能和功能需求。\n",
    "\n",
    "## 5. Scrapy：强大的爬虫框架\n",
    "\n",
    "### 5.1 简介\n",
    "\n",
    "Scrapy是一个用Python编写的应用框架，用于编写爬虫，从网站中提取结构化数据。它提供了一整套工具，使得构建和扩展爬虫变得简单高效。\n",
    "\n",
    "### 5.2 安装\n",
    "\n",
    "使用pip安装Scrapy：\n",
    "\n",
    "```bash\n",
    "pip install scrapy\n",
    "```\n",
    "\n",
    "### 5.3 Scrapy的核心概念\n",
    "\n",
    "1. Spider：定义如何爬取特定网站的类。\n",
    "2. Item：定义要抓取的数据结构。\n",
    "3. Item Pipeline：处理抓取到的数据。\n",
    "4. Selector：从网页中提取数据的机制。\n",
    "5. Downloader Middleware：处理请求和响应。\n",
    "6. Spider Middleware：处理spider的输入和输出。\n",
    "\n",
    "### 5.4 创建Scrapy项目\n",
    "\n",
    "```bash\n",
    "scrapy startproject myproject\n",
    "cd myproject\n",
    "scrapy genspider example example.com\n",
    "```\n",
    "\n",
    "这将创建一个新的Scrapy项目和一个名为\"example\"的spider。\n",
    "\n",
    "### 5.5 定义Item\n",
    "\n",
    "在`items.py`中定义要抓取的数据结构：\n",
    "\n",
    "```python\n",
    "import scrapy\n",
    "\n",
    "class MyProjectItem(scrapy.Item):\n",
    "    title = scrapy.Field()\n",
    "    author = scrapy.Field()\n",
    "    content = scrapy.Field()\n",
    "```\n",
    "\n",
    "### 5.6 编写Spider\n",
    "\n",
    "编辑`spiders/example.py`：\n",
    "\n",
    "```python\n",
    "import scrapy\n",
    "from myproject.items import MyProjectItem\n",
    "\n",
    "class ExampleSpider(scrapy.Spider):\n",
    "    name = 'example'\n",
    "    allowed_domains = ['example.com']\n",
    "    start_urls = ['http://example.com/']\n",
    "\n",
    "    def parse(self, response):\n",
    "        for article in response.css('article'):\n",
    "            item = MyProjectItem()\n",
    "            item['title'] = article.css('h2::text').get()\n",
    "            item['author'] = article.css('span.author::text').get()\n",
    "            item['content'] = article.css('div.content::text').get()\n",
    "            yield item\n",
    "\n",
    "        next_page = response.css('a.next-page::attr(href)').get()\n",
    "        if next_page is not None:\n",
    "            yield response.follow(next_page, self.parse)\n",
    "```\n",
    "\n",
    "### 5.7 运行Spider\n",
    "\n",
    "```bash\n",
    "scrapy crawl example\n",
    "```\n",
    "\n",
    "### 5.8 使用Item Pipeline\n",
    "\n",
    "在`pipelines.py`中定义如何处理抓取到的数据：\n",
    "\n",
    "```python\n",
    "import json\n",
    "\n",
    "class JsonWriterPipeline:\n",
    "    def open_spider(self, spider):\n",
    "        self.file = open('items.json', 'w')\n",
    "\n",
    "    def close_spider(self, spider):\n",
    "        self.file.close()\n",
    "\n",
    "    def process_item(self, item, spider):\n",
    "        line = json.dumps(dict(item)) + \"\\n\"\n",
    "        self.file.write(line)\n",
    "        return item\n",
    "```\n",
    "\n",
    "在`settings.py`中启用pipeline：\n",
    "\n",
    "```python\n",
    "ITEM_PIPELINES = {\n",
    "    'myproject.pipelines.JsonWriterPipeline': 300,\n",
    "}\n",
    "```\n",
    "\n",
    "### 5.9 使用Selector\n",
    "\n",
    "Scrapy使用Selector来提取数据：\n",
    "\n",
    "```python\n",
    "# CSS选择器\n",
    "title = response.css('h1::text').get()\n",
    "\n",
    "# XPath选择器\n",
    "author = response.xpath('//span[@class=\"author\"]/text()').get()\n",
    "\n",
    "# 正则表达式\n",
    "date = response.css('span.date::text').re_first(r'(\\d{4}-\\d{2}-\\d{2})')\n",
    "```\n",
    "\n",
    "### 5.10 处理表单和登录\n",
    "\n",
    "```python\n",
    "class LoginSpider(scrapy.Spider):\n",
    "    name = 'login'\n",
    "    start_urls = ['http://example.com/login']\n",
    "\n",
    "    def parse(self, response):\n",
    "        return scrapy.FormRequest.from_response(\n",
    "            response,\n",
    "            formdata={'username': 'john', 'password': 'secret'},\n",
    "            callback=self.after_login\n",
    "        )\n",
    "\n",
    "    def after_login(self, response):\n",
    "        # 检查登录是否成功\n",
    "        if \"authentication failed\" in response.body:\n",
    "            self.logger.error(\"Login failed\")\n",
    "            return\n",
    "        # 继续爬取受保护的页面\n",
    "```\n",
    "\n",
    "### 5.11 中间件的使用\n",
    "\n",
    "在`middlewares.py`中定义中间件：\n",
    "\n",
    "```python\n",
    "from scrapy import signals\n",
    "\n",
    "class MyProjectDownloaderMiddleware:\n",
    "    @classmethod\n",
    "    def from_crawler(cls, crawler):\n",
    "        s = cls()\n",
    "        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)\n",
    "        return s\n",
    "\n",
    "    def process_request(self, request, spider):\n",
    "        # 在请求被下载器处理之前调用\n",
    "        return None\n",
    "\n",
    "    def process_response(self, request, response, spider):\n",
    "        # 在响应被传给爬虫处理之前调用\n",
    "        return response\n",
    "\n",
    "    def process_exception(self, request, exception, spider):\n",
    "        # 当下载器或处理器抛出异常时调用\n",
    "        pass\n",
    "\n",
    "    def spider_opened(self, spider):\n",
    "        spider.logger.info('Spider opened: %s' % spider.name)\n",
    "```\n",
    "\n",
    "在`settings.py`中启用中间件：\n",
    "\n",
    "```python\n",
    "DOWNLOADER_MIDDLEWARES = {\n",
    "    'myproject.middlewares.MyProjectDownloaderMiddleware': 543,\n",
    "}\n",
    "```\n",
    "\n",
    "### 5.12 并发和性能控制\n",
    "\n",
    "Scrapy默认使用异步网络库Twisted来处理并发。你可以通过以下设置来控制并发：\n",
    "\n",
    "```python\n",
    "# settings.py\n",
    "\n",
    "# 同时处理的请求数量\n",
    "CONCURRENT_REQUESTS = 16\n",
    "\n",
    "# 对单个域名的并发请求数\n",
    "CONCURRENT_REQUESTS_PER_DOMAIN = 8\n",
    "\n",
    "# 对单个IP的并发请求数\n",
    "CONCURRENT_REQUESTS_PER_IP = 0\n",
    "\n",
    "# 下载延迟\n",
    "DOWNLOAD_DELAY = 3\n",
    "```\n",
    "\n",
    "### 5.13 数据导出\n",
    "\n",
    "Scrapy支持多种数据导出格式：\n",
    "\n",
    "```bash\n",
    "scrapy crawl example -o items.json\n",
    "scrapy crawl example -o items.csv\n",
    "scrapy crawl example -o items.xml\n",
    "```\n",
    "\n",
    "你也可以在`settings.py`中配置自定义的导出：\n",
    "\n",
    "```python\n",
    "FEED_EXPORTERS = {\n",
    "    'csv': 'myproject.exporters.CustomCsvItemExporter',\n",
    "}\n",
    "```\n",
    "\n",
    "### 5.14 分布式爬虫\n",
    "\n",
    "Scrapy可以与Scrapy-Redis结合使用，实现分布式爬虫：\n",
    "\n",
    "1. 安装Scrapy-Redis：\n",
    "\n",
    "   ```bash\n",
    "   pip install scrapy-redis\n",
    "   ```\n",
    "\n",
    "2. 修改`settings.py`：\n",
    "\n",
    "   ```python\n",
    "   SCHEDULER = \"scrapy_redis.scheduler.Scheduler\"\n",
    "   DUPEFILTER_CLASS = \"scrapy_redis.dupefilter.RFPDupeFilter\"\n",
    "   REDIS_URL = 'redis://localhost:6379'\n",
    "   ```\n",
    "\n",
    "3. 修改Spider继承自`RedisSpider`：\n",
    "\n",
    "   ```python\n",
    "   from scrapy_redis.spiders import RedisSpider\n",
    "   \n",
    "   class MySpider(RedisSpider):\n",
    "       name = 'myspider'\n",
    "       \n",
    "       def parse(self, response):\n",
    "           # 解析逻辑\n",
    "   ```\n",
    "\n",
    "### 5.15 处理JavaScript渲染的页面\n",
    "\n",
    "对于JavaScript渲染的页面，Scrapy可以与Splash集成：\n",
    "\n",
    "1. 安装Scrapy-Splash：\n",
    "\n",
    "   ```bash\n",
    "   pip install scrapy-splash\n",
    "   ```\n",
    "\n",
    "2. 在`settings.py`中配置Splash：\n",
    "\n",
    "   ```python\n",
    "   SPLASH_URL = 'http://localhost:8050'\n",
    "   DOWNLOADER_MIDDLEWARES = {\n",
    "       'scrapy_splash.SplashCookiesMiddleware': 723,\n",
    "       'scrapy_splash.SplashMiddleware': 725,\n",
    "       'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,\n",
    "   }\n",
    "   SPIDER_MIDDLEWARES = {\n",
    "       'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,\n",
    "   }\n",
    "   DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'\n",
    "   ```\n",
    "\n",
    "3. 在Spider中使用Splash：\n",
    "\n",
    "   ```python\n",
    "   from scrapy_splash import SplashRequest\n",
    "   \n",
    "   class MySpider(scrapy.Spider):\n",
    "       name = 'myspider'\n",
    "       \n",
    "       def start_requests(self):\n",
    "           yield SplashRequest(url=\"https://example.com\", callback=self.parse)\n",
    "       \n",
    "       def parse(self, response):\n",
    "           # 解析逻辑\n",
    "   ```\n",
    "\n",
    "### 5.16 处理验证码\n",
    "\n",
    "对于需要处理验证码的网站，可以结合OCR库（如Tesseract）或验证码识别服务：\n",
    "\n",
    "```python\n",
    "import pytesseract\n",
    "from PIL import Image\n",
    "import io\n",
    "\n",
    "class CaptchaSpider(scrapy.Spider):\n",
    "    name = 'captcha_spider'\n",
    "    \n",
    "    def parse(self, response):\n",
    "        # 假设验证码图片的URL在某个img标签的src属性中\n",
    "        captcha_url = response.css('img#captcha::attr(src)').get()\n",
    "        yield scrapy.Request(captcha_url, callback=self.solve_captcha)\n",
    "    \n",
    "    def solve_captcha(self, response):\n",
    "        img = Image.open(io.BytesIO(response.body))\n",
    "        captcha_text = pytesseract.image_to_string(img)\n",
    "        \n",
    "        # 使用识别出的验证码文本提交表单\n",
    "        yield FormRequest.from_response(\n",
    "            response,\n",
    "            formdata={'captcha': captcha_text},\n",
    "            callback=self.after_captcha\n",
    "        )\n",
    "    \n",
    "    def after_captcha(self, response):\n",
    "        # 处理提交验证码后的响应\n",
    "        pass\n",
    "```\n",
    "\n",
    "### 5.17 最佳实践和性能优化\n",
    "\n",
    "1. 使用Item Loaders处理数据：\n",
    "\n",
    "   ```python\n",
    "   from scrapy.loader import ItemLoader\n",
    "   from scrapy.loader.processors import TakeFirst, MapCompose, Join\n",
    "   \n",
    "   class ProductLoader(ItemLoader):\n",
    "       default_output_processor = TakeFirst()\n",
    "       name_in = MapCompose(str.strip)\n",
    "       price_in = MapCompose(str.strip, lambda x: x.split('$')[-1])\n",
    "       description_out = Join()\n",
    "   \n",
    "   # 在spider中使用\n",
    "   loader = ProductLoader(item=ProductItem(), response=response)\n",
    "   loader.add_css('name', 'h1.product-name::text')\n",
    "   loader.add_xpath('price', '//span[@class=\"price\"]/text()')\n",
    "   loader.add_css('description', 'div.description::text')\n",
    "   yield loader.load_item()\n",
    "   ```\n",
    "\n",
    "2. 使用`yield`而不是`return`来返回数据，保持内存使用效率。\n",
    "\n",
    "3. 适当设置`CONCURRENT_REQUESTS`、`DOWNLOAD_DELAY`等参数，避免对目标网站造成过大压力。\n",
    "\n",
    "4. 实现自定义的去重机制，特别是对于大规模爬虫项目。\n",
    "\n",
    "5. 使用`JOBDIR`设置来启用爬虫的暂停和恢复功能：\n",
    "\n",
    "   ```bash\n",
    "   scrapy crawl myspider -s JOBDIR=crawls/myspider-1\n",
    "   ```\n",
    "\n",
    "6. 对于大规模项目，考虑使用Scrapyd进行爬虫部署和管理。\n",
    "\n",
    "7. 使用Scrapy的日志功能进行调试和监控：\n",
    "\n",
    "   ```python\n",
    "   self.logger.info('Processed item %s', item['name'])\n",
    "   ```\n",
    "\n",
    "8. 对于需要频繁更新的数据，考虑实现增量爬取策略。\n",
    "\n",
    "9. 使用`scrapy-user-agents`或自定义中间件轮换User-Agent。\n",
    "\n",
    "10. 对于敏感操作，实现IP代理池：\n",
    "\n",
    "    ```python\n",
    "    class ProxyMiddleware:\n",
    "        def process_request(self, request, spider):\n",
    "            request.meta['proxy'] = \"http://proxy.example.com:8050\"\n",
    "    ```\n",
    "\n",
    "### 5.18 Scrapy vs 其他爬虫方法\n",
    "\n",
    "- 优点：\n",
    "  1. 高度可定制和可扩展\n",
    "  2. 内置并发处理\n",
    "  3. 提供完整的爬虫开发框架\n",
    "  4. 丰富的中间件和插件生态系统\n",
    "\n",
    "- 缺点：\n",
    "  1. 学习曲线相对陡峭\n",
    "  2. 对于简单任务可能显得过于复杂\n",
    "  3. 默认不支持JavaScript渲染（需要额外配置）\n",
    "\n",
    "Scrapy特别适合大型、复杂的爬虫项目，尤其是那些需要高度定制和优化的场景。对于小型或一次性的爬虫任务，使用Requests+BeautifulSoup可能更为简单直接。\n",
    "\n",
    "### 结语\n",
    "\n",
    "本教程详细介绍了Python爬虫开发中最常用的几个库：Requests、Beautiful Soup、lxml、Selenium和Scrapy。每个库都有其特定的用途和优势：\n",
    "\n",
    "- Requests适用于简单的HTTP请求处理。\n",
    "- Beautiful Soup适合解析结构不太规范的HTML。\n",
    "- lxml在处理大量结构化数据时表现出色。\n",
    "- Selenium能够处理需要浏览器交互的复杂场景。\n",
    "- Scrapy则是构建大规模、高性能爬虫的理想选择。\n",
    "\n",
    "在实际的爬虫项目中，这些工具往往会结合使用。例如，你可能会使用Scrapy作为主框架，结合lxml进行高效的HTML解析，在需要处理JavaScript渲染的页面时使用Selenium。\n",
    "\n",
    "选择合适的工具组合取决于你的具体需求，包括目标网站的复杂度、数据量、性能要求等因素。随着经验的积累，你将能够更好地评估每个项目的需求，并选择最合适的工具和策略。\n",
    "\n",
    "记住，网络爬虫应该遵守道德和法律规范，包括尊重网站的robots.txt文件、控制请求频率、以及遵守网站的使用条款。负责任的爬虫开发不仅能帮助你避免法律问题，还能维护良好的互联网生态。"
   ],
   "id": "3c452a946ab46aaf"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "outputs": [],
   "execution_count": null,
   "source": "",
   "id": "8ba1f8e32ef6322f"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
